The present invention relates to digital video signal processing, and more particularly to architectures and methods for digital cameras.
Imaging and video capabilities have become the trend in consumer electronics. Digital cameras, digital camcorders, and video cellular phones are common, and many other new gadgets are evolving in the market. Advances in large resolution CCD/CMOS sensors coupled with the availability of low-power digital signal processors (DSPs) has led to the development of digital cameras with both high resolution image and short audio/visual clip capabilities. The high resolution (e.g., sensor with a 2560×1920 pixel array) provides quality offered by traditional film cameras.
FIG. 3a is a typical functional block diagram for digital camera control and image processing (the “image pipeline”). The automatic focus, automatic exposure, and automatic white balancing are referred to as the 3A functions; and the image processing includes functions such as color filter array (CFA) interpolation, gamma correction, white balancing, color space conversion, and JPEG/MPEG compression/decompression (JPEG for single images and MPEG for video clips). Note that the typical color CCD consists of a rectangular array of photosites (pixels) with each photosite covered by a filter (the CFA): typically, red, green, or blue. In the commonly-used Bayer pattern CFA one-half of the photosites are green, one-quarter are red, and one-quarter are blue.
Typical digital cameras provide a capture mode with full resolution image or audio/visual clip processing plus compression and storage, a preview mode with lower resolution processing for immediate display of a sequence of images (video clip), and a playback mode for displaying stored images or audio/visual clips.
Images captured by handheld cameras occasionally suffer from image blur when there is not enough light in the scene and/or when high zoom factors are used. This is a significant problem for today's consumer digital cameras and camera phones. A solution for eliminating blur is required to obtain high quality images.
Image deblurring has been an active research area during the last few decades with numerous algorithms published in the literature. For example, M. R. Banham and A. K. Katsaggelos. “Digital Image Restoration,” IEEE Signal Processing Magazine, Vol. 14, Issue 2, pp.24-41, March 1997; D. Kundur and D. Hatzinakos. “Blind Image Deconvolution,” IEEE Signal Processing Magazine, Vol. 13, Issue 3, pp.43-64, May 1996; and J. Biemond, R. L. Lagendijk, and R. M. Mersereau. “Iterative Methods of Image Deblurring,” Proc. of IEEE, Vol. 78, Issue 5, pp. 856-883, May 1990.
Classical image deblurring algorithms require the knowledge of the blur function. The blur function is used to design a filter that removes the blur from the image. There have been several published algorithms that address the problem of estimating the blur function. They mostly involve assuming a parametric form for the blur and then estimating its parameters from the image. For example, J. Biemond, R. L. Lagendijk, and R. M. Mersereau. “Iterative Methods of Image Deblurring,” Proc. of IEEE, Vol. 78, Issue 5, pp. 856-883, May 1990; and M. M Chang, A. M. Tekalp, and A. T. Erdem. “Blur Identification Using the Bispectrum,” IEEE Trans. Signal Processing, Vol. 39, Issue 10, pp. 2323-2325, Oct. 1991.
The deblurring model using the blur function presumes the blurred pixel at (m,n), denoted pB(m,n), is given by a two-dimensional convolution of the unblurred image p(m, n) with the blurred image of a single point source; that is, convolution with the point spread function (PSF) or blur function. Denoting the blur function by h(m,n):pB(m,n)=∫support of h p(m−j, n−k)h(j,k) dj dkwhere integration over the blur function variables is interpreted as interpolated summations for sub-pixel values. Then transforming (2-D DFT) to the frequency domain formally gives a deconvolution:P(j,k)=PB(j,k)/H(j,k)where P(j,k) is the 2-D DFT transform of p(m,n), PB(j,k) is the 2-D DFT transform of pB(m,n), and H(j,k) is the 2-D DFT transform of h(m,n). However, this raises division by 0 problems, so a regularization term α would typically be included, such as in a constrained least squares approach:P(j,k)=PB(j,k) H(j,k)*/(|H(j,k)|2+α)A final inverse transform (2-D IDFT) recovers the deblurred image.
An example of a typical parametric blur function comes from a linear blur presumption. A motion vector (Vx, Vy) is estimated, and the blurred pixels are a simple averaging along the motion path such as:pB(m,n)=∫0<t<1 p(m−tNVx, n−tNVy) dtwhere the common motion vector has been scaled to the time interval of the image capture (and blur motion). Of course, this is just a two-dimensional convolution of the unblurred image p(m, n) with the following blur function:h(m,n)=∫0<t<1 δ(m−tNVx) δ(n−tNVy) dtwhich would be interpreted in terms of interpolations for the non-integer pixel locations and Kronecker deltas for the Dirac deltas.
For the handheld camera motion blur case, a typical assumption is to use the linear blur function and estimate two parameters: length and direction of the motion vector. These parameters are estimated from the log power spectrum of the image by locating the minimum points, which are assumed to correspond to the zeros of the blur transfer function. Unfortunately, this approach does not provide an accurate estimate of the blur because hand motion is not really on a linear path. The mismatch between the actual motion and the linear motion model creates numerous artifacts in the deblurred image. FIGS. 2a-2b show deblurring an image using a linear blur function estimate. Note that there are significant artifacts in the deblurred image shown in FIG. 2b. Such a result is not acceptable for consumer quality products.